Crowdsourced network measurements: Benefits and best practices

Network measurements are of high importance both for the operation of networks and for the design and evaluation of new management mechanisms. Therefore, several approaches exist for running network measurements, ranging from analyzing live traffic traces from campus or Internet Service Provider (ISP) networks to performing active measurements on distributed testbeds, e.g., PlanetLab, or involving volunteers. However, each method falls short, offering only a partial view of the network. For instance, the scope of passive traffic traces is limited to an ISP’s network and customers’ habits, whereas active measurements might be biased by the population or node location involved. To complement these techniques, we propose to use (commercial) crowdsourcing platforms for network measurements. They permit a controllable, diverse and realistic view of the Internet and provide better control than do measurements with voluntary participants. In this study, we compare crowdsourcing with traditional measurement techniques, describe possible pitfalls and limitations, and present best practices to overcome these issues. The contribution of this paper is a guideline for researchers to understand when and how to exploit crowdsourcing for network measurements

Impact of the Number of Votes on the Reliability and Validity of Subjective Speech Quality Assessment in the Crowdsourcing Approach

The subjective quality of transmitted speech is traditionally assessed in a controlled laboratory environment according to ITU-T Rec. P.800. In turn, with crowdsourcing, crowdworkers participate in a subjective online experiment using their own listening device, and in their own working environment. Despite such less controllable conditions, the increased use of crowdsourcing micro-task platforms for quality assessment tasks has pushed a high demand for standardized methods, resulting in ITU-T Rec. P.808. This work investigates the impact of the number of judgments on the reliability and the validity of quality ratings collected through crowdsourcing-based speech quality assessments, as an input to ITU-T Rec. P.808 . Three crowdsourcing experiments on different platforms were conducted to evaluate the overall quality of three different speech datasets, using the Absolute Category Rating procedure. For each dataset, the Mean Opinion Scores (MOS) are calculated using differing numbers of crowdsourcing judgements. Then the results are compared to MOS values collected in a standard laboratory experiment, to assess the validity of crowdsourcing approach as a function of number of votes. In addition, the reliability of the average scores is analyzed by checking inter-rater reliability, gain in certainty, and the confidence of the MOS. The results provide a suggestion on the required number of votes per condition, and allow to model its impact on validity and reliability.Comment: This paper has been accepted for publication in the 2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX

Towards speech quality assessment using a crowdsourcing approach: evaluation of standardized methods

Subjective speech quality assessment has traditionally been carried out in laboratory environments under controlled conditions. With the advent of crowdsourcing platforms tasks, which need human intelligence, can be resolved by crowd workers over the Internet. Crowdsourcing also offers a new paradigm for speech quality assessment, promising higher ecological validity of the quality judgments at the expense of potentially lower reliability. This paper compares laboratory-based and crowdsourcing-based speech quality assessments in terms of comparability of results and efficiency. For this purpose, three pairs of listening-only tests have been carried out using three different crowdsourcing platforms and following the ITU-T Recommendation P.808. In each test, listeners judge the overall quality of the speech sample following the Absolute Category Rating procedure. We compare the results of the crowdsourcing approach with the results of standard laboratory tests performed according to the ITU-T Recommendation P.800. Results show that in most cases, both paradigms lead to comparable results. Notable differences are discussed with respect to their sources, and conclusions are drawn that establish practical guidelines for crowdsourcing-based speech quality assessment

iMobilePhysics: Seamless Learning durch Experimente mit Smartphones & Tablets in Physik

Der Beitrag erläutert ein Nutzungskonzept für Smartphones und Tablets als physikalische Experimentiermittel (iMobilePhysics, kurz: iMP) und ordnet zwei Realisierungsbeispiele dieses Konzept für Schule und Hochschule an Hand von Designkriterien in den Rahmen des Mobile Assisted Seamless Learning (MSL) ein. Zu den Beispielen werden Studien zur Lernwirksamkeit und Akzeptanz der medienbasierten Lehransätze dargestellt. Die Ergebnisse liefern den Begründungsrahmen, mobiles Experimentieren mit Smartphone und Tablets zur Unterstützung von MSL in die schulische und universitäre Lehre, insbesondere in die Lehramtsausbildung einzubinden. 12.08.2016 | Michael Hirth, Jochen Kuhn, Andreas Müller, Matthias Rohs & Pascal Klein (Kaiserslautern, Genf

Crowdsourcing Technology to Support Academic Research

No abstract available

CO-independent modification of K+ channels by tricarbonyldichlororuthenium(II) dimer (CORM-2)

Although toxic when inhaled in high concentrations, the gas carbon monoxide (CO) is endogenously produced in mammals, and various beneficial effects are reported. For potential medicinal applications and studying the molecular processes underlying the pharmacological action of CO, so-called CO-releasing molecules (CORMs), such as tricabonyldichlororuthenium(II) dimer (CORM-2), have been developed and widely used. Yet, it is not readily discriminated whether an observed effect of a CORM is caused by the released CO gas, the CORM itself, or any of its intermediate or final breakdown products. Focusing on Ca2+- and voltage-dependent K+ channels (KCa1.1) and voltage-gated K+ channels (Kv1.5, Kv11.1) relevant for cardiac safety pharmacology, we demonstrate that, in most cases, the functional impacts of CORM-2 on these channels are not mediated by CO. Instead, when dissolved in aqueous solutions, CORM-2 has the propensity of forming Ru(CO)2 adducts, preferentially to histidine residues, as demonstrated with synthetic peptides using mass-spectrometry analysis. For KCa1.1 channels we show that H365 and H394 in the cytosolic gating ring structure are affected by CORM-2. For Kv11.1 channels (hERG1) the extracellularly accessible histidines H578 and H587 are CORM-2 targets. The strong CO-independent action of CORM-2 on Kv11.1 and Kv1.5 channels can be completely abolished when CORM-2 is applied in the presence of an excess of free histidine or human serum albumin; cysteine and methionine are further potential targets. Off-site effects similar to those reported here for CORM-2 are found for CORM-3, another ruthenium-based CORM, but are diminished when using iron-based CORM-S1 and absent for manganese-based CORM-EDE1

Unemployment and online labor

Online labor markets experienced a rapid growth in recent years. They allow for long-distance transactions and offer workers access to a potentially ‘global’ pool of labor demand. As such, they bear the potential to act as a substitute for shrinking local income opportunities. Using detailed U.S. data from a large online labor platform for microtasks, we study how local unemployment affects participation and work intensity online. We find that, at the extensive margin, an increase in commuting zone level unemployment is associated with more individuals joining the platform and becoming active in fulfilling tasks. At the intensive margin, our results show that with higher unemployment rates, online labor supply becomes more elastic. These results are driven by a decrease of the reservation wage during standard working hours. Finally, the effects are transient and do not translate to a permanent increase in platform participation by incumbent users. Our findings highlight that many workers consider online labor markets as a substitute to offline work for generating income, especially in periods of low local labor demand. However, the evidence also suggests that, despite their potential to attract workers, online markets for microtasks are currently not viable as a long run alternative for most workers

Survey of Web-based Crowdsourcing Frameworks for Subjective Quality Assessment

The popularity of the crowdsourcing for performing various tasks online increased significantly in the past few years. The low cost and flexibility of crowdsourcing, in particular, attracted researchers in the field of subjective multimedia evaluations and Quality of Experience (QoE). Since online assessment of multimedia content is challenging, several dedicated frameworks were created to aid in the designing of the tests, including the support of the testing methodologies like ACR, DCR, and PC, setting up the tasks, training sessions, screening of the subjects, and storage of the resulted data. In this paper, we focus on the web-based frameworks for multimedia quality assessments that support commonly used crowdsourcing platforms such as Amazon Mechanical Turk and Microworkers. We provide a detailed overview of the crowdsourcing frameworks and evaluate them to aid researchers in the field of QoE assessment in the selection of frameworks and crowdsourcing platforms that are adequate for their experiments